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Abstract 

INScore is an open source framework for the design of 
interactive, augmented, live music scores. Augmented 
music scores are graphic spaces providing representa¬ 
tion, composition and manipulation of heterogeneous 
and arbitrary music objects (music scores but also im¬ 
ages, text, signals...), both in the graphic and time do¬ 
mains. INScore includes also a dynamic system for 
the representation of the music performance, consid¬ 
ered as a specific sound or gesture instance of the score, 
and viewed as signals. It integrates an event based in¬ 
teraction mechanism that opens the door to original 
uses and designs, transforming a score as a user in¬ 
terface or allowing a score self-modification based on 
temporal events. This paper presents the system fea¬ 
tures, the underlying formalisms, and introduces the 
OSC based scripting language. 
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1 Introduction 

Music notation has a long history and evolved 
through ages. From the ancient neumes to the 
contemporary music notation, the western culture 
is rich of the many ways explored to represent the 
music. From symbolic or prescriptive notations 
to pure graphic representation, the music score 
has always been in constant interaction with the 
creative and artistic process. 

However, although the music representations 
have exploded with the advent of computer music 
[Dannenberg, 1993; Hewlett and Selfridge-Field, 
2001], the music score, intended to the performer, 
didn’t evolved in proportion to the new music 
forms. In particular, there is a significant gap 
between interactive music and the static way it is 
usually notated: a performer has generally a tra¬ 
ditional paper score, plus a computer screen dis¬ 
playing a number or a letter to indicate the state 


of the interaction system. New needs in terms of 
music representation emerge of this context. 

In the domain of electro-acoustic music, ana¬ 
lytic scores - music scores made a posteriori , be¬ 
come common tools for the musicologists but have 
little support from the existing computer music 
software, apart the approach proposed for years 
by the Acousmograph [Geslin and Lefevre, 2004]. 

In the music pedagogy domain and based on a 
mirror metaphor, experiments have been made to 
extend the music score in order to provide feed¬ 
back to students learning and practicing a tradi¬ 
tional music instruments [Fober et ah, 2007]. This 
approach was based on an extended music score, 
supporting various annotations, including perfor¬ 
mance representations based on the audio signal, 
but the system was limited by a monophonic score 
centered approach and a static design of the per¬ 
formance representation. 

Today, new technologies allow for real-time in¬ 
teraction and processing of musical, sound and 
gestural information. But the symbolic dimen¬ 
sion of the music is generally excluded from the 
interaction scheme. 

INScore has been designed in answer to these 
observations. It is an open source framework 1 for 
the design of interactive, augmented, live music 
scores. It extends the traditional music score to 
arbitrary heterogeneous graphic objects: it sup¬ 
ports symbolic music notation (Guido [Hoos et ah, 
1998] or MusicXML [Good, 2001] format), text 
(utf8 encoded or html format), images (jpeg, tiff, 
gif, png, bmp), vectorial graphics (custom basic 
shapes or SVG), video files, as well as an original 
performance representation system [Fober et ah, 
2010b]. 

Each component of an augmented score has a 

x http://inscore.sf.net 



graphic and temporal dimension and can be ad¬ 
dressed in both the graphic and temporal space. 
A simple formalism, is used to draw relations be¬ 
tween the graphic and time space and to repre¬ 
sent the time relations of any score components 
in the graphic space. Based on the graphic and 
time space segmentation, the formalism relies on 
simple mathematical relations between segments 
and on relations compositions. We talk of time 
synchronization in the graphic domain [Fober et 
ah, 2010a] to refer to this specific feature. 

INScore is a message driven system that makes 
use of the Open Sound Control [OSC] 2 format. 
This design opens the door to remote control and 
to interaction using any OSC capable application 
or device. In addition, it includes interaction fea¬ 
tures provided at score component level by the 
way of watchable events. 

All these characteristics make INScore a 
unique system in the landscape of music notation 
or of multi-media presentation. The next section 
gives details about the system features, including 
the underlying formalisms. Next the OSC API is 
introduced with a special emphasis on how it is 
turned into a scripting language. The last section 
gives an overview of the system architecture and 
dependencies before some directions are presented 
for future work. 

2 Features 

Table 1 gives the typology of the graphic resources 
supported by the system. All the score elements 
have graphic properties (position, scale, rotation, 
color, etc.) and time properties as well (date and 
duration). 

INScore provides a message based API to cre¬ 
ate and control the score elements, both in the 
graphic and time spaces. The Open Sound Con¬ 
trol [OSC] protocol is used as basis format for 
these messages, described in section 3. 

2.1 Time to graphic relations 

All the elements of a score have a time dimension 
and the system is able to graphically represent the 
time relationships of these elements. INScore is 
using segmentations and mappings to achieve this 
time synchronization in the graphic domain. 

The segmentation of a score element is a set 
of segments included in this element. A segment 

2 http://opensoundcontrol.org/ 


may be viewed as a portion of an element. It 
could be a 2D graphic segment (a rectangle), a 
time interval, a text section, or any segment de¬ 
scribing a part of the considered element in its 
local coordinates space. 

Mappings are mathematical relations between 
segmentations. A composition operation is used 
to relate the graphic space of two arbitrary score 
elements, using their relation to the time space. 
Table 2 lists the segmentations and mappings 
used by the different component types. Mappings 
are indicated by arrows (<H>). Note that the ar¬ 
rows link segments of different types. Segmen¬ 
tations and mappings in italic are automatically 
computed by the system, those in bold are user 
defined. 

Note that for music scores, an intermediate 
time segmentation, the wrapped time , is necessary 
to catch repeated sections and jumps (to sign, to 
coda, etc.). 
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Figure 1: Graphic segments of a score and its 
performance displayed using colors and annotated 
with the corresponding time segments. 


Figure 1 shows segments defined on a score and 
an image, annotated with the corresponding time 
segments. Synchronizing the image to the score 
will stretch and align the graphic segments corre¬ 
sponding to similar time segments as illustrated 
by figure 2. 

Description of the image graphic to time rela¬ 
tion is given in table 3 as a set of pairs including 
a graphic segment defined as 2 intervals on the x 
and y axis, and a time segment defined as 2 ratio¬ 
nal expressing musical time (where 1 represents 
a whole note). 

















Symbolic music description 

Guido Music Notation and MusicXML formats 

Text 

plain text or html (utf8) 

Images 

jpg, gif, tiff, png, bmp 

Vectorial graphics 

line, rectangle, ellipse, polygon, curve, SVG code 

Video files 

using the phonon plugin 

Performance representations 

see section 2.2 


Table 1: Graphic resources supported by the system. 


type 

segmentations and mappings required 

Symbolic music description 
Text 
Images 
Vectorial graphics 
Performance representations 

graphic wrapped time time 

graphic ^ text time 

graphic ^ pixel ^ time 
graphic vectorial o- time 

graphic frame time 


Table 2: Segmentations and mappings for each component type 
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Figure 2: Time synchronization of the segments 
displayed in figure 1. 


( [0, 67[ [0, 86[ ) ( [0/2, 1/2[ ) 

( [67, 113[ [0, 86[ ) ( [1/2, 1/1[) 

( [113, 153[ [0, 86[ ) ( [1/1, 5/4[ ) 

( [153, 190[ [0, 86[ ) ( [5/4, 3/2[ ) 

( [190, 235[ [0, 86[ ) ( [3/2, 4/2[ ) 


Table 3: A mapping described as a set of relations 
between graphic and time segments. 

2.2 Performance representation 

Music performance representation is based on sig¬ 
nals, whether audio or gestural signals. To pro¬ 
vide a flexible and extensible system, the graphic 
representation of a signal is viewed as a graphic 
signal , i.e. as a composite signal made of: 

• a y coordinate signal 

• a thickness signal h 

• a color signal c 


Such a composite signal (see figure 3) includes 
all the information required to be drawn without 
additional computation. 



t 

Figure 3: A composite graphic signal at time t. 


The following examples show some simple rep¬ 
resentations defined using this model. 

2.2.1 Pitch representation 

Represents notes pitches on the y-axis using the 
fundamental frequency (figure 4). 


Figure 4: Pitch representation. 

The corresponding graphic signal is expressed 
as: 

9 = Sfo / h / k c 

where Sfo : fundamental frequency 




























kt : a constant thickness signal 
k c : a constant color signal 

2.2.2 Art iculat ions 

Makes use of the signal RMS values to control the 
graphic thickness (figure 5). The corresponding 



Figure 5: Articulations. 


where Sfo : fundamental frequency 
SrmsO ' fO RMS values 
k c 0 : a constant color signal 
Next we build the graphic for the harmonic 1: 

9 1 = S f0 / Srmsl T SrmsO / k c \ 

Srmsi : harmonic 1 RMS values 
k c 1 : a constant color signal 
Next, the graphic for the harmonic 2: 


graphic signal is expressed as: 


9 2 — Sfo/ S rms 2 + S rms i H- S rm sO / k c 2 


9 — ky / ^rms / kc 


where k y : signal y constant 
S rrns • RMS signal 
k c : a constant color signal 

2.2.3 Pitch and articulation combined 

Makes use of the fundamental frequency and RMS 
values to draw articulations shifted by the pitches 
(figure 6). 
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Figure 6: Pitch and articulation combined. 

The corresponding graphic signal is expressed 
as: 

9 S f0 / Srms / k c 

where Sfo : fundamental frequency 
Srms • RMS signal 
k c : a constant color signal 

2.2.4 Pitch and harmonics combined 

Combines the fundamental frequency to the first 
harmonics RMS values (figure 7). Each harmonic 
has a different color. 



Figure 7: Pitch and harmonics combined. 

The graphic signal is described in several steps. 
First, we build the fundamental frequency graphic 
as above (see section 2.2.3) : 

gO — SfQ / SrmsO / k c 0 


S r ms2 • harmonic 2 RMS values 
k c 2 : a constant color signal 

etc. 

Finally, the graphic signals are combined into a 
parallel graphic signal: 

9 = #2 / gl / gO 

2.3 Interaction 

INScore is a message driven system that makes 
use of Open Sound Control [OSC] format. It in¬ 
cludes interaction features provided at individ¬ 
ual score component level by the way of watch- 
able events , which are typical events available in 
a graphic environment, extended in the temporal 
domain. The list of supported events is given in 
table 4. 


mouse events 

time events 

misc. 

mouse enter 
mouse leave 
mouse down 
mouse up 

mouse move 

double click 

time enter 
time leave 

new element 
(at scene level) 


Table 4: Typology of watchable events. 

Events are associated to user defined messages 
that are triggered by the event occurrence. The 
message includes a destination address (INScore 
by default) that supports a url-like specification, 
allowing to target any IP host on any UDP port. 
The message associated to mouse events may use 
predefined variables, instantiated at event time 
with the current mouse position or the current 
position time. 





































Application 


This simple event based mechanism makes easy 
to describe for example an intelligent cursor i.e. 
an arbitrary object that is synchronized to the 
score and that turns the page to the next or pre¬ 
vious one, depending on the time zone it enters. 

3 INScore messages 

The INScore API is an OSC messages API. The 
general format is illustrated in figure 8: it con¬ 
sists in an address followed by a message string, 
followed by parameters. 




Figure 10: The set message. 


Figure 8: General format of a message. 

The address may be viewed as an object 
pointer, the message string as a method name 
and the parameters as the method parameters. 
For example, the message: 

/ITL/scene/score color 255 128 40 150 
may be viewed as the following method call: 
score->color(255 128 40 150) 

3.1 Address space 

The OSC address space is strictly organized like 
the internal score representation, which is a tree 
with 4 depths levels (figure 9). Messages can be 
sent to any level of the hierarchy. 

The first level is the application level: a static 
node with a fixed address that is also used to 
discriminate incoming messages. An application 
contains several scenes, which corresponds to dif¬ 
ferent windows and different scores. Each scene 
contains components and two static nodes: a sync 
node for the components synchronization and a 
signal node, that may be viewed as a special 
folder containing signals. The name in blue are 
user defined, those in black are static reserved 
names. 

3.2 Message strings 

This section gives some of the main messages sup¬ 
ported by all the score components. The list is far 
from being exhaustive, it is intended to show ex¬ 
amples of the system API. 

Score components are created and/or modified 
using a set message (figure 10) that takes the 


object type as argument, followed by type specific 
parameters. 

Messages given in figure 11 are used to set the 
objects graphic properties. 



Figure 11: Graphic space management. 

Messages given in figure 12 set the time proper¬ 
ties. Time is encoded as rational values represent¬ 
ing music time (where 1 is a whole note). Graphic 
space and time management messages have rela¬ 
tive positioning forms: the message string pre¬ 
fixed with ’d\ 

All the messages that modify the system state 
have a counterpart get form illustrated by figure 
13. 

Synchronization between components is based 
on a master / slave scheme. It is described by 
a message addressed to the static sync node as 
illustrated by figure 14. syncmode indicates how 
to align the slave component to its master: hori¬ 
zontal and/or vertical stretch, etc. 

Interactive features are available by request¬ 
ing to a component to watch an event using 
















































Figure 12: Time management. 



Figure 14: Synchronization message: the second 
form (2) removes the synchronization from the 
slave object. 


^watch)— what — oscMsg 


Figure 13: Querying the system state. 

a message like figure 15, where what indicates 
the event (mouseUp, mouseDown, mouseEnter, 
mouseLeave, timeEnter, timeLeave) and OSCMsg 
represents any valid OSC message including the 
address part. 

The example in figure 16 creates a simple score, 
do some scaling, creates a red cursor and synchro¬ 
nizes it to the score with a vertical stretch to the 
score height. Output is presented by figure 17. 

3.3 INScore scripts 

Actually, the example given in figure 16) is mak¬ 
ing use of the file format of a score, which consists 
in a list of textual OSC messages , separated by a 
semi-colon. This textual form is particularly suit¬ 
able to be used as a scripting language and addi¬ 
tional support is provided in the form of variables 
and javascript and/or lua support. 

3.3.1 Variables 

INScore scripts supports variables declarations 
in the form illustrated by figure 18. Variables may 
be used in place of any message parameter pre¬ 
fixed using a $ sign. A variable must be declared 
before being used. 

3.3.2 Scripting support 

INScore scripts may include javascript sections, 
delimited by <? javascript and ?> The javascript 
code is evaluated at parse time. It is expected 
to produce valid INScore messages as output. 
These messages are then expanded in place of the 
javascript code. 

Variables declared before the javascript section 
are exported to the javascript environment, which 
make these variables usable in both contexts. 


Figure 15: Watching events. 

Similarly, INScore may support the lua lan¬ 
guage in sections delimited by <?lua and ?>. 
By default, lua support is not embedded in the 
INScore binary distribution. The engine needs 
to be compiled with the appropriate options to 
support lua. 

4 Architecture 

INScore is both a shared library and a standalone 
score viewer application without user interface. 
Some insight of the system architecture is given 
in this section for a better understanding and an 
optimal use of the system. The general architec¬ 
ture is a Model View Controller [MVC] designed 
to handle OSC message streams. 

4.1 The MVC Model 

The MVC architecture is illustrated in figure 19. 
Modification of the model state is achieved by in¬ 
coming OSC messages or by the library C/C++ 
API, that is actually also message based. An 
OSC message is packaged into an internal mes¬ 
sages representation and stacked on a lock-free 
fifo stack. These operations are synchronous to 
the incoming OSC stream and to the library API 
call. 

On a second step, messages are popped from 
the stack by a Controler that takes in charge 

/ITL/scene/score set ’gnuC Mg e f d] ; ; 
/ITL/scene/score scale 5.0; 

/ITL/scene/cursor set *rect J 0.01 0.1; 
/ITL/scene/cursor color 255 0 0 150; 
/ITL/scene/sync ; cursor ; ; score’ ; 

Figure 16: INScore sample script. 
































Implementation 



Figure 17: Sample script output. 

<-> 


<5 


ident 


-Q.nt32y~ 


float32 


string^- ^ 


Figure 18: Variable declaration. 

address decoding and message passing to the cor¬ 
responding object of the model. The final opera¬ 
tion concerns the view update. An Updater is in 
charge of producing a view of the model, which is 
currently based on the Qt Framework. 

This second step of the model modification 
scheme is asynchronous: it is processed on a reg¬ 
ular time base. 

4.2 Dependencies 

The INScore project depends on external open 
source libraries: 

• the Qt framework 3 providing the graphic 
layer and platform independence. 

• the GuidoEngine 4 for music score layout 

• the GuidoQt static library, actually part of 
the Guido library project 

• the oscpack library 5 for OSC support 

• and optionally: 

— the MusicXML library 6 to support the 
MusicXML format. 

— the v8 javascript engine 7 to support 
javascript in INScore script files. 

— the lua engine 8 to support lua in 
INScore script files. 

3 http://qt.nokia.com/ 

4 http://guidolib.sourceforge.net 

5 http://www.rossbencina.com/code/oscpack 

6 http://libmusicxml.sourceforge.net 

7 http://code.google.com/p/v8/ 

8 http://www.lua.org/ 
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Figure 19: An overview of the MVC architecture 


The GuidoEngine, the GuidoQt and oscpack li¬ 
braries are required to compile INScore. 

Binary packages are distributed for Ubuntu 
Linux, Mac OS and Windows. Detailed instruc¬ 
tions are included in the project repository for 
compiling. 

5 Future work 

Interactive features described in section 2.3 result 
from an experimental approach. The formaliza¬ 
tion and extension of these features are planned. 

Time synchronization features are based on a 
continuous music time. Supporting other time 
representations like the Allen relations [Allen, 
1983] is part of the future work. 

Due to its dynamic nature, INScore is particu¬ 
larly suitable to interactive music, i.e. where parts 
of the music score could be computed in real-time 
by an interaction system. It could be particularly 
interesting to provide a musically meaningful vi¬ 
sualization of the interaction system state; exten¬ 
sions in this direction are also planned. 
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